Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source: files (s3 and local) #653

Merged
merged 24 commits into from
Dec 3, 2024
Merged

Source: files (s3 and local) #653

merged 24 commits into from
Dec 3, 2024

Conversation

tim-quix
Copy link
Contributor

@tim-quix tim-quix commented Nov 26, 2024 •
edited
Loading

Not 100% done yet but here is a fairly refined draft. Just a little more cleanup to do and a tiny bit more work left on Azure, and then documentation assuming we like the approach.

Currently extends FileSource (which itself had some minor adjustments to better accommodate extension).

@tim-quix tim-quix added the connector Issues updating Sinks or Sources label Nov 26, 2024
@tim-quix tim-quix marked this pull request as draft November 26, 2024 01:43
@tim-quix tim-quix changed the title Source: blob store (AWS and Azure) Source: blob stores (AWS and Azure) Nov 26, 2024
gwaramadze
gwaramadze reviewed Nov 27, 2024
View reviewed changes
docs/connectors/sources/aws-s3-file-source.md Outdated Show resolved Hide resolved
quixstreams/sources/community/file/origins/s3.py Outdated Show resolved Hide resolved
quixstreams/sources/community/file/origins/local.py Outdated Show resolved Hide resolved
quixstreams/sources/community/file/origins/local.py Show resolved Hide resolved
quixstreams/sources/community/file/origins/local.py Outdated Show resolved Hide resolved
quixstreams/sources/community/file/origins/s3.py Outdated Show resolved Hide resolved
quixstreams/sources/community/file/origins/base.py Outdated Show resolved Hide resolved
@tim-quix
Copy link
Contributor Author

tim-quix commented Nov 27, 2024 •
edited
Loading

TODO:

  • maybe add a simple file download queue?
  • look over Tomas's updated replay timestamp logic and see if changes are needed
  • double-check non-code stuff (adding doc paths, package stuff, etc)

@tim-quix tim-quix changed the title Source: blob stores (AWS and Azure) Source: external files (AWS and Azure) Nov 27, 2024
@tim-quix tim-quix changed the title Source: external files (AWS and Azure) Source: external files (S3 and Azure) Nov 27, 2024
tim-quix and others added 18 commits December 2, 2024 13:53
@tim-quix
initial commit, try to fix pickling error
c66e2a5
@tim-quix
fixed issue around pickling, need to simplify file gathering and part…
6389fcd 
…ition counting
@tim-quix
created more shared methods and unified classes more
fa4ad96
@tim-quix
a few more tweaks
9045a00
@tim-quix
move everything to file source
6fee37d
@tim-quix
remove previous blob_store addition
5dd5d98
@tim-quix
clean up some things, add pagination handling to azure
dae3fc6
@tim-quix
add docs, rename some stuff
211b3cd
@tim-quix
clean up schema doc section
e23fc57
@tim-quix
clean up compressor
71792df
@tim-quix
tiny doc tweaks
4c3e6c3
@tim-quix
changes based on code review
8cff599
@tim-quix @gwaramadze
Update docs/connectors/sources/aws-s3-file-source.md
8a699ac 
Co-authored-by: Remy Gwaramadze <gwaramadze@users.noreply.github.com>
@tim-quix
change bucket variable name
c26a217
@tim-quix
update local file docs
bb12e07
@tim-quix
add azure docs
07af800
@tim-quix
clean up azure stuff and other small tweaks
72cba51
@tim-quix
clean up after second round of code review
f581e28
@tim-quix tim-quix changed the title Source: external files (S3 and Azure) Source: files (s3 and local) Dec 3, 2024
@tim-quix tim-quix marked this pull request as ready for review December 3, 2024 03:26
@tim-quix
Copy link
Contributor Author

tim-quix commented Dec 3, 2024

This is ready to merge, depending on final thoughts on the client stuff for S3.

@daniil-quix daniil-quix merged commit 69bf48b into main Dec 3, 2024
3 checks passed
@daniil-quix daniil-quix deleted the source/blob_store branch December 3, 2024 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daniil-quix daniil-quix daniil-quix left review comments

@gwaramadze gwaramadze gwaramadze approved these changes

Assignees
No one assigned
Labels
connector Issues updating Sinks or Sources
Projects
Quix Streams - Roadmap
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tim-quix @gwaramadze @daniil-quix